Testing models with a nonminimal Higgs sector through the decay t — > q + WZ 
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We study the contribution of the charged Higgs boson to the rare decay of the top quark t — > 
q + WZ (q = d, s, b) in models with Higgs sectors that include doublets and triplets. Higgs doublets 
are needed to couple a charged Higgs with quarks, whereas the Higgs triplets are required to generate 
the nonstandard vertex HWZ at the tree level. It is found that within a model that respects the 
custodial SU (2) c symmetry and avoids flavor-changing neutral current (FCNC) by imposing discrete 
symmetries, the decay mode t — ► b+ WZ can reach a branching ratio (BR) of order 10 -2 , whereas 
the decay modes t —* [d, s) + WZ, can reach a similar BR in models where FCNC are suppressed 
by flavor symmetries. 
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(\) ' INTRODUCTION 

The mass of the top quark, which is larger than any other fermion mass in the standard model (SM)and almost 
t— I , as large as its scale of electroweak symmetry breaking (EWSB), cannot be explained within the SM M. This has 
originated speculations about the possible relationship between the top quark and the nature of the mechanism 
responsible for EWSB. Several models have been proposed, where such large mass can be accommodated or plays 
a significant role. In the supersymmetric (SUSY) extensions of the SM ||, the large value of the top quark mass 
can drive the radiative breaking of the electroweak symmetry; furthermore within the context of SUSY grand unified 
theories (GUT's) the fermions of the third-family can be accommodated in scheemes where their masses arise from a 
single Yukawa term On the other hand, in some top-condensate (TC) models p| it is postulated that new strong 
t-H , interactions bind the heavy top quark into a composite Higgs scenario. 

' From a more phenomenological point of view, it is also intriguing to notice that the top quark decay seems to be 
dominated by the SM mode (t — > bW), not only within the SM but also in theories beyond it, which makes the top 
> quark decay width almost insensitive to the presence of new physics; unless the scale of new physics is lighter than the 
Oh! top quark mass itself, such that new states can appear in its decays. This is the case, for instance, in the general two 
Higgs doublet model (THDM-III) || , where the flavor changing mode t — > c + h can be important for a light Higgs 
boson (h), or in SUSY models with light stop quark and neutralinos, in whose case the decay t — > i + \° can also be 
*iZ< relevant. But in general, the rare decays of the top quark have undetectable branching ratios (BR's); for instance, 
the flavor-changing neutral current (FCNC) rare decays t — > cV (V = 7, Z, g) have a very small BR in the SM, of the 
order 10 -11 ||, and are out of reach of present and future colliders. A similar result is obtained in several extensions 
of the SM; for instance in the THDM-II, minimal SUSY extensions of the SM (MSSM) and left -right models, to 
mention some cases |7j. 

The rare decay t — > qWZ (q = b, s, d), may be above the threshold for the production of a real WZ state, provided 
that m t > m q + mw + mz ■ The possibilities to satisfy this relation depend on the final state q and the precise value 
of the top quark mass, which according to the Particle Data Group Q is m t = 173.8 ± 5.2 GeV. For the case when 
q = b, the top quark mass must satisfy m t > 176.1 ± 0.5 GeV, where the uncertainty on the right-hand side is mostly 
due to the ambiguity in the bottom quark mass, thus t — > bWZ can occur on-shell only if mt takes its upper allowed 
value (at la). However, if q — d or s, the decays t — > qWZ can occur even when m t takes its central value. 

The value of BR(t — > bWZ) predicted in SM is 5.4 x 1CP 7 ||, which is beyond the sensitivity of Tevatron Run II or 
even CERN Large Hadron Collider (LHC); thus its observation would truly imply the presence of new physics. For 
q = d,s the SM result is even smaller, since the amplitude is supressed by the Cabibbo-Kobayashi-Maskawa (CKM) 
matrix elements Vt q . On the other hand, the decay mode t — > qWZ can proceed through an intermediate charged 
Higgs boson that couples to both tq and WZ currents, and thus can be used to test the couplings of Higgs sectors 



beyond the SM |10| 



The construction of extensions of the SM Higgs sector must satisfy the constraints impossed by the successful 
phenomenological relation p = m^/rnr^ cos 8 W = 1, which also measures the ratio between the neutral and charged 
current couplings strength. At tree level this relation is satisfied naturally in models that include only Higgs doublets, 
but in more general scenarios, there could be tree level contributions to p — 1. Since the vertex HWZ arises at tree 
level only for Higgs bosons lying in representations higher than the usual SM Higgs doublet, there could be violations 
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of the constraints impossed by the p parameter. However, tree-level deviations of the electroweak p parameter from 
unity can be avoided by arranging the nondoublet fields and the vacuum expectation values (V.E.V's) of their neutral 
members, so that a custodial SU(2) C symmetry is maintained |11|. On the other hand, a generic coupling of the 
charged Higgs boson with fermions may be associated with the possible appearence of FCNC in the Higgs- Yukawa 
sector. FCNC are automatically absent in the minimal SM with one Higgs doublet, however in multiscalar models 
large FCNC can appear if each quark flavor couples to more than one Higgs doublet |L2fl . FCNC can be avoided either 
by impossing some ad hoc discrete symmetry to the Yukawa Lagrangian, i.e., by coupling each type of fermion only to 
one Higgs doublet; or by using flavor symmetries. The former case is used in the so-called two Higgs doublet models 
I and II, whereas the last one is associated with model III, here FCNC is only suppressed by some ansatz for the 
Yukawa matrices, for instance the Li-Sher one: {Yq)ij = y/rniirij/mw, whose phenomenology was studied in jlslTlf. 



In this Rapid Communication we shall consider, in a very general setting, the contribution of a charged Higgs boson 
to the decay t — > qWZ , and present the results in terms of two factors that parametrize the doublet-triplet mixing 
and the nonminimal Yukawa couplings, respectively Then, we discuss the values that these parameters can take for 
specific extensions of the SM, when the constraints from both the custodial symmetry and FCNC are satisfied, and 
present the predicted values for BR(t — » qWZ). 



THE DECAY T -» QWZ 

We are interested in studying the contribution of charged Higgs boson to the rare decay of the top quark t — > qWZ 
(q = d,s,b), within the context of models with extended Higgs sector that include additional Higgs doublets and 
triplets. The charged Higgs will be assumed to be the lightest charged mass eigenstate that results from the general 
mixing of doublets and triplets in the charged sector. ^ 

Higgs doublets are needed in order to couple the charged Higgs with quarks; the vertex tqH^ will be written as 
follows 

— -j= rjtq cos a [(mt cot (3 + m q tan j3) + (mt cot (3 — m q tan f3)js] , (1) 

2 v 2mw 

which can be considered as a modification of the result obtained for the Yukawa sector of the general THDM, where 
cos a is included to account for the doblet-triplet mixing; tan j3 is the ratio of the V.E.V.'s of the two scalar doublets. 
The charged Higgs coupling to the quarks is also determined by the parameters r] tq , which is equal to the CKM mixing 
matrix only for model-II, i.e., = V tq ; however, in the general case (THDM-III), one can have rj^ 1 > V tq ||. 

On the other hand, we require a representation higher than the doublet, in order to obtain a sizeable coupling 
HWZ at tree level, which is written as 

~c^; sina5 -- (2) 

In order to evaluate the decay t — > qWZ, we shall write a general amplitude to describe the contribution of the 
intermediate charged Higgs, neglecting the SM contribution, which is a good approximation since the corresponding 
BR is very suppressed. To calculate the amplitude one also needs to take into account the finite width of the 
intermediate charged Higgs boson with momentum pn , mass mn , and width Th, for this we shall use the relativistic 
Breit-Wigner form of the propagator in the unitary gauge. Then, the amplitude can be written in general as 

M = A [u(p q )(a + b l5 )u(p t )] ( ~\ a ) [g^e*i] , (3) 

where rhn = fna + (i/ty^H'i a, b, and A are constants related to the parameters a and r\ t q previously mentioned 

a = m t cot [3 + m q tan (3, 

b = m t cot j3 — m q tan f3, (4) 

A = — -= rj tq cos a sin a. 

2V2 cos 9 W 



1 This is justified by our explicit analysis of the Higgs potential for several models with Higgs doublets and triplets, which will 
be presented elsewhere. 
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To calculate the partial decay width, we shall perform a numerical integration of the expression for the squared 
amplitude, over the standard three-body phase space, namely 



T(t -► qWZ) = 



' 1 1 ' M I 2 dsdt. 



(2tt) 3 32m? 



(5) 



A4 | 2 denotes the squared amplitude, averaged over initial spins and summed over final polarizations, it has the form 



The integration limits are 



and 



\A | 2 [(a 2 + b 2 )(m 2 + m 2 - s) + (a 2 - b 2 )2m t m q ] 
' = (s-m 2 H ) 2 + m 2 H T 2 H 

(m w + m z ) 2 < s < (m t - m q ) 2 , 
t~ <t< t + , 



2 9 \ 2 

s - m w - m% 
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where 



+ m%- ^-[(s + m 2 -m 2 )(s + m| -m 2 w )T A 1/2 (s, m 2 , m 2 )A 1/2 (s, m|, m 2 ^)], 



(6) 

(7) 
(8) 
(9) 



and A(x, y, z) = (x + y — z) 2 — Axy. 

The branching ratio for this decay is obtained as the ratio of Eq. (||) to the total width of the top quark, which will 
include the modes t — > qW and t — * qH; the expressions for the widths are 



r(t -> qW + ) = 



Gftu^ 
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and 



T(t -> qH^ 
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w m t 



T) tq cos 2 a [a 2 [(m t + m q ) 2 - m 2 H ] + b 2 [(m t - m q ) 2 - m 2 H ]] 



(11) 



On the other hand, the Higgs width will include the fermionic decays into H —> cs and H — > tv t \ adding them we 
obtain 



g 2 m H 



3271171^ 



cos 2 a [3rj cs (ml cot 2 (3 + m 2 tan 2 (3) + m 2 tan 2 0\ 



as well as the bosonic mode H — » VFZ 



r(F+^^+Z)-^-^sin 2 a 
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(12) 



(13) 



RESULTS AND CONCLUSIONS 



In order to present the results for the mode t — > bWZ , i.e., q = b, we shall assume that the top quark mass takes its 
upper allowed value, and will consider a Yukawa sector similar to the model-II, in whose case the factor rj tb is equal to 
the CKM matrix element Vtb (— 1); the results are shown for two values of tan /? (2, and rrit/mb) which are acceptable 
for GUT- Yukawa unification. For the factor cos a sin a, which is part of the constant A, we shall consider first the 
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value i, which corresponds to the maximum value that can be expected to arise in an scenario where the custodial 
symmetry is respected, for instance in a model with one Higgs doublet and two Higgs triplets of hypercharges and 2, 
respectively, where one can align the V.E.V.'s to respect the custodial symmetry and obtaining p — 1 p[ . Q On the 
other hand, to consider a model without a custodial symmetry, we take the value sin a = 0.04, which corresponds to 
the maximum value that is allowed by the experimental error in the p parameter With all these considerations, 
we shown in Fig. [I] our results for the BR of the decay t — ► bWZ; we notice that it can reach a maximum value of 
order 1.78 x 10" 2 . 

For the decays into the light quarks, still working within the framework of model II, we obtain a very suppressed 
result, where we are taking now the central value for the top quark mass, namely, for t — > sWZ we get a maximum 
value for the BR of order 1.95 x 10~ 6 for sin a cos a = i and tan/3 ~ 2; for t — » dWZ we get results even smaller 
and thus uninteresting. On the other hand, if we consider a model with a Yukawa sector of the type THDM-III, the 
coupling of the charged Higgs with the quarks is not determined by the CKM mixing matrix, then the couplings tdH~ 
and tsH~ may not be suppresed. Although in model III there can be dangerous FCNC, it happens that such effects 
have not been tested by top quark decays, and thus can give large and detectable effects 0. For the parameter a 
we take the same values of the previous case, assuming also rjt s — r\td we get a maximum value for the BR of order 
1.31 x 10~ 3 for both t -> (d, s) + WZ, as shown in Fig. |. 

We conclude from our results that there exist a region of parameters where it is possible to obtain a large BR for 
the decay t — > bWZ. Moreover, for m# = 162 to itih — 182 GeV, cos a sin a = i, and tan/3 = m t /mb we obtain 
a BR larger than the one predicted by the SM. Furthermore, the maximum value for the BR, of order 10 -2 , seems 
factible to be detected at the future CERN LHC, where about 10 8 top quark pairs could be produced, and one would 
have 10 6 events of interest with only one top quark decaying rarely. If we also include the decays of the W and Z 
into leptonic modes, to allow a clear signal, one would end with about 1.3 x 10 events, which is interesting enough to 
perform a future detailed study of backgrounds; however this is beyond the scope of present work. On the other hand, 
we observe from Fig. ^ that even within models without a custodial symmetry with sin a = 0.04, it is possible to get 
BR for the decay t — > sWZ larger than the SM result, or the result obtained within models where sin a cos a = |, 
depending on the value of tan/3; in some cases it can reach a BR of order 1.31 x 10 -3 . 

In conclusion, we find that the decay t — > qWZ is sensitive to the contribution of new physics, in particular from a 
charged Higgs boson, which makes this mode an interesting arena for testing physics beyond the SM. 
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2 Although our framework is similar to the one of Ref. in our case we are allowing full mixing between all the scalar 

multiplets of the model, which allows us to have charged and neutral Higgs bosons that couple simultaneously to both fermion 
and gauge boson pairs. 
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FIG. 1. Branching ratio for the decay t — > bWZ, as 
a function of vhh for cos a sin a — i (upper curves) and 
sin a = 0.04 (lower curves); and for tan/3 = (solid 
curves) and tan/3 = 2 (dashed curves); it is also assumed 
that rjtq = Vt q . The line indicate the SM prediction. 

FIG. 2. Branching ratio for the decay t — > sWZ, as a func- 
tion of rriH for cos a sin a = i (upper dashed and solid curve) 
and sin a = 0.04 (middle curve); and for tan/3 = ^ (solid 
curve) and tan/3 = 2 (dashed curves); taking also ijt q = 1. 
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